![](\"banner.png\"/)
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Comparing travel behavior during 2019 and 2020 in the New York City Taxi and Limousine Commission trip records\n",
"\n",
"by Nika Karen Espiritu and Jasper Pangan"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"EXECUTIVE SUMMARY
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"New York declared their first COVID-19 case on March 1, 2020. Since then, it has turned local economy upside down and spelled the deaths of thousands of people. For the safety of its people, the government has issued strict stay-at-home orders. For the New York taxi drivers, this has endangered their livelihoods and families. In this paper, we ask how travel patterns and behaviors have changed from 2019 to 2020 with the pandemic in place. To answer this question, we gathered data from the New York City (NYC) Taxi and Limousine Commission (TLC), pre-processed it, and conducted an exploratory data analysis (EDA) to provide descriptive analytics. The dataset considered in this study spans from January 1, 2019 until June 30, 2020, covering until Phase 2 of New York’s reopening plans. From our EDA, we found little change in routes and in travel times of New Yorkers, but instead we found that New Yorkers strictly complied with the stay-at-home measures enacted by the government, thus signifying a drop in the volume of taxi transactions. People usually travelled alone in pre-pandemic times, but this phenomenon increased during the pandemic. We also found that cashless transactions were banned by legislators, and people are experiencing the impacts of this legislation. With the reopening of New York, we are now seeing a small increase in volume of transactions, and this should help NYC taxi drivers."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"INTRODUCTION
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"China announced the first case of coronavirus on December 31, 2019 [[1]](#ref1). By March 1, 2020, New York City (NYC) confirmed its first case [[2]](#ref2). Unlike other parts of the world affected by the virus, the government of New York City initially brushed off the crisis, with the NYC Health Commissioner even stating that the risk of New Yorkers were “low” [[3]](#ref3). A month later, it quickly became the epicenter of the US as it had more cases than China [[4]](#ref4). \n",
"\n",
"Government response has worked double time to balance safety of the people as well as health of the economy. However, the impact of the pandemic still stands to this day: unemployment has surged, businesses and restaurants have shut down, and lives have been lost [[5]](#ref5). \n",
"\n",
"In this paper, we zoom in to see how the pandemic has affected transactions made with New York taxi drivers and compare this to their figures before the pandemic. **With New York City being the epicenter of the COVID-19 pandemic, how has travel behavior changed through the course of 2020 compared to pre-pandemic times?** \n",
"\n",
"To answer this question, we extracted and cleaned the NYC TLC data and conducted an exploratory data analysis (EDA) to answer various questions regarding the pandemic’s effect on mobility, economy, and society. The whole process is done by a Dask cluster."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"DATA
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Data of trips taken by taxis and vehicles in New York City were retrieved from the NYC Taxi and Limousine Commission (TLC) Trip Record Data found in the Registry of Open Data on Amazon Web Services (AWS) S3 bucket [[6]](#ref6).\n",
"\n",
"Data available in the NYC TLC applicable for this research is only from January 1, 2019 to June 30, 2020. This corresponds to $9.36$ GB worth of data. After preprocessing, this corresponds to $80,810,133$ transactions in 2019 and $15,859,906$ transactions in 2020."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"METHODOLOGY
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To uncover changes in mobility of New Yorkers due to COVID-19, a total of $96,670,039$ transactions were retrieved from AWS Registry of Open Data [[6]](#ref6). The general workflow for providing descriptive analytics as shown in [Figure 1](#fig1) involves the following steps:\n",
"\n",
"1. Establishment of Dask Cluster\n",
"2. Data Extraction\n",
"3. Data Processing\n",
"4. Exploratory Data Analysis\n",
"5. Descriptive Analytics\n",
"\n",
"Each step of the workflow will be discussed in the succeeding sections.\n",
"\n",
"**Figure 1**. Workflow for descriptive analytics of NYC Taxi Trips.\n",
"![](\"workflow.png\"/)
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### 1. Establishment of Dask Cluster\n",
"\n",
"We first set up our `Dask` scheduler and workers through Amazon Web Services’ Elastic Compute Cloud (AWS EC2) web service, as we will be dealing with $9.36$ GB of data thus the need for distributed computing. To run the succeeding codes, we have assigned Jojie as our client. "
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"ExecuteTime": {
"end_time": "2020-11-30T10:26:31.652170Z",
"start_time": "2020-11-30T10:26:29.184409Z"
}
},
"outputs": [],
"source": [
"client = Client('72.44.51.212:8786')"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"ExecuteTime": {
"end_time": "2020-11-30T10:26:57.221313Z",
"start_time": "2020-11-30T10:26:56.750914Z"
}
},
"outputs": [
{
"data": {
"text/html": [
"\n",
"Client\n", "
| \n",
"\n",
"Cluster\n", "
| \n",
"
1:Credit card,
2:Cash,
3:No charge,
4:Dispute| \n", "|passenger_count|Number of passengers| \n", "|trip*|Number of transaction|\n", "|year*|Year of transaction| \n", "|month*|Month of transaction|\n", "|day*|Day of transaction| \n", "|hour*|Hour of transaction| \n", " \n", "**derived field* \n", " \n", "Separate Dask dataframes were built for hourly, daily, and monthly aggregation using helper functions.\n", "\n", "The conduct of the exploratory data analysis and its structure are found in the succeeding section." ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "ExecuteTime": { "end_time": "2020-11-30T10:27:27.647941Z", "start_time": "2020-11-30T10:27:27.566581Z" } }, "outputs": [], "source": [ "def clean_data(df, year):\n", " \"\"\"Return a processed data frame\"\"\"\n", " # Add month of transaction\n", " df['month'] = (df.tpep_pickup_datetime\n", " .astype('M8[us]')\n", " .dt.month)\n", " # Add day of transaction\n", " df['day'] = (df.tpep_pickup_datetime\n", " .astype('M8[us]')\n", " .dt.day)\n", " # Add year of transaction\n", " df['year'] = (df.tpep_pickup_datetime\n", " .astype('M8[us]')\n", " .dt.year)\n", " # Add hour of transaction\n", " df['hour'] = (df.tpep_pickup_datetime\n", " .astype('M8[us]')\n", " .dt.hour)\n", " # Add trip count per transaction\n", " df['trip'] = 1\n", "\n", " # Filter data\n", " df = df[(df['total_amount'] > 0) &\n", " (~df['passenger_count'].isna()) &\n", " (df['passenger_count'] > 0) &\n", " (df['PULocationID'] < 264) &\n", " (df['DOLocationID'] < 264) &\n", " (df['trip_distance'] > 0) &\n", " (df['year'] == year)]\n", "\n", " if year == 2020:\n", " df = df[df['month'] < 7]\n", "\n", " columns = ['month', 'day', 'year', 'hour', 'trip',\n", " 'payment_type', 'passenger_count', 'PULocationID',\n", " 'DOLocationID']\n", " return df[columns]\n", "\n", "\n", "df_clean_2019 = clean_data(df_2019, 2019)\n", "df_clean_2020 = clean_data(df_2020, 2020)" ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "ExecuteTime": { "end_time": "2020-11-30T10:27:27.866474Z", "start_time": "2020-11-30T10:27:27.649764Z" } }, "outputs": [], "source": [ "def agg_daily(df):\n", " \"\"\"Return a persisted data frame aggregated daily\"\"\"\n", " return (df.groupby(['month', 'day', 'payment_type', \n", " 'passenger_count', 'PULocationID','DOLocationID'])\n", " .trip\n", " .count()\n", " .reset_index()\n", " .persist())\n", "\n", "df_agg_daily_2020 = agg_daily(df_clean_2020)\n", "df_agg_daily_2019 = agg_daily(df_clean_2019) " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 4. Exploratory Data Analysis" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "According to the NYC TLC, demand for the New York taxi went down by 90%, placing 83% of drivers in a tough spot wherein they have either struggled to afford food or could not afford food [[10]](#ref10). At the same time, COVID-19 cases have continuously increased, as shown in [Figure 2](#fig2). This has also led to nearly 40% of drivers either contracting COVID-19 or living with someone who tested positive for the virus [[10]](#ref10). With the drop in demand, what does this look like? By the numbers, how did COVID-19 affect mobility of New Yorkers? With the phased opening and decreasing case rates, are things looking better for NYC taxi drivers?" ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "ExecuteTime": { "end_time": "2020-11-30T10:27:29.358815Z", "start_time": "2020-11-30T10:27:27.937292Z" } }, "outputs": [], "source": [ "path = ('https://raw.githubusercontent.com/fedhere/PUI2015_EC/master/'\n", " 'mam1612_EC/nyc-zip-code-tabulation-areas-polygons.geojson')\n", "with urlopen(path) as response:\n", " data = json.load(response)" ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "ExecuteTime": { "end_time": "2020-11-30T10:27:29.375380Z", "start_time": "2020-11-30T10:27:29.360729Z" } }, "outputs": [], "source": [ "df_covid = pd.read_csv('data-by-modzcta.csv')\n" ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "ExecuteTime": { "end_time": "2020-11-30T10:27:30.146775Z", "start_time": "2020-11-30T10:27:29.379980Z" } }, "outputs": [ { "data": { "text/markdown": [ "**Figure 2**. Total COVID-19 cases by zip code as of November 26, 2020." ], "text/plain": [ "
RESULTS AND DISCUSSION
"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {
"ExecuteTime": {
"end_time": "2020-11-30T10:38:44.167377Z",
"start_time": "2020-11-30T10:27:30.150205Z"
}
},
"outputs": [],
"source": [
"def agg_monthly(df, year):\n",
" \"\"\"Return a pandas data frame aggregated monthly\"\"\"\n",
" df = df.groupby('month')['trip'].sum().compute().reset_index()\n",
" df['date'] = pd.to_datetime(year + '/' + \n",
" df.month.astype(str) + '/' \n",
" + '1')\n",
" return df\n",
"\n",
"viz = (agg_monthly(df_agg_daily_2019, '2019')\n",
" .append(agg_monthly(df_agg_daily_2020, '2020')))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We provide a preliminary overview on the impact of coronavirus in [Figure 3](#fig3)."
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {
"ExecuteTime": {
"end_time": "2020-11-30T10:38:45.740524Z",
"start_time": "2020-11-30T10:38:44.170927Z"
}
},
"outputs": [
{
"data": {
"text/markdown": [
"**Figure 3**. Total monthly taxi transactions from January 1, 2019 to June 30, 2020."
],
"text/plain": [
"Make this Notebook Trusted to load map: File -> Trust Notebook